ESQL: Type conflict resolution in unmapped-fields load by GalLalouche · Pull Request #143693 · elastic/elasticsearch

GalLalouche · 2026-03-05T15:54:36Z

This PR adds support for resolving type conflicts with unmapped fields. Similar to how union types work, this is resolved using explicit casts, except we have an additional constraint that potentially unmapped fields are treated as KEYWORD.
In particular, if a field is Partially Unmapped and when it is mapped it's a Non-Keyword (or PUNK, as per @mouhc1ne), we have a type conflict, which may itself be conflicted if the field is mapped to different fields (in addition to the unmapped KEYWORD).

Resolves #141912.
Resolves #142004.

PotentiallyUnmappedKeywordEsField (used for fields loaded from _source when unmapped_fields="load") had isAggregatable=true, which caused isPushableFieldAttribute to short-circuit past the SearchStats check and push filters down to Lucene. On shards where the field is not indexed, the Lucene query returns no results instead of letting the compute engine evaluate the filter on _source-loaded values. Guard isPushableFieldAttribute against PotentiallyUnmappedKeywordEsField so these fields are always filtered in the compute engine. Co-authored-by: Cursor <cursoragent@cursor.com>

The isPushableFieldAttribute fix (rejecting PotentiallyUnmappedKeywordEsField) already covers sort pushdown since PushTopNToSource uses the same method. This test verifies correct sort order when a field is mapped in one index but unmapped in another under unmapped_fields="load". Co-authored-by: Cursor <cursoragent@cursor.com>

x-pack/plugin/esql/qa/testFixtures/src/main/resources/unmapped-load.csv-spec

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

GalLalouche · 2026-03-23T13:42:11Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/unmapped-load.csv-spec

 ;

+partiallyUnmappedNonKeyword
+required_capability: optional_fields_v2


GalLalouche · 2026-03-23T13:44:50Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/unmapped-load.csv-spec

+required_capability: index_metadata_field
+
+SET unmapped_fields="load"\;
+FROM sample_data, no_mapping_sample_data METADATA _index


GalLalouche · 2026-03-23T13:49:09Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/unmapped-load.csv-spec

+;
+
+#######################
+# Type conflict tests #


GalLalouche · 2026-03-23T13:49:56Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/unmapped-load.csv-spec

+# Type conflict tests #
+#######################
+
+typeConflictKeywordUnmappedCastToKeyword


GalLalouche · 2026-03-24T17:05:46Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

                        var convert = new ToDateNanos(f.source(), f, context.configuration());
                        imf.types().forEach(type -> typeResolutions(f, convert, type, imf, typeResolutions));
-                        var resolvedField = ResolveUnionTypes.resolvedMultiTypeEsField(f, typeResolutions);
+                        var resolvedField = ResolveUnionTypes.resolvedMultiTypeEsField(f, typeResolutions, null);


So while analyzing this, I uncovered an underlying pre-existing union type bug! #144870.

It looks like this is actually fine to be null (I added an assertion), but the other one isn't.

GalLalouche · 2026-03-25T12:10:10Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

                            indexToConversionExpressions,
-                            fa.field().getTimeSeriesFieldType()
+                            fa.field().getTimeSeriesFieldType(),
+                            null // FIXME(gal, NOCOMMIT) This basically a copy constructor, shouldn't be null; write a test to flush it out.


This is actually fine, due to the behavior mentioned in #144870. I've added an assertion and comment instead.

GalLalouche · 2026-03-25T20:51:32Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

This part though, is a bug! If some fields are unmapped and load is on, then not all types are actually dates! Fixed and added tests.

x-pack/plugin/esql/qa/testFixtures/src/main/resources/unmapped-load.csv-spec

GalLalouche · 2026-03-26T11:18:41Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/unmapped-load.csv-spec

Done, except TS. Now it's a race to see who finishes first :)

GalLalouche · 2026-03-26T14:12:20Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

                            indexToConversionExpressions,
-                            fa.field().getTimeSeriesFieldType()
+                            fa.field().getTimeSeriesFieldType(),
+                            null


Added an assert and a comment why this should be null.

GalLalouche · 2026-03-26T14:14:31Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/IndexResolver.java

-            if (indexMappingHashToDuplicateCount.compute(response.getIndexMappingHash(), (k, v) -> v == null ? 1 : v + 1) > 1) {
-                continue;
-            }
+            boolean isNew = indexMappingHashToDuplicateCount.compute(


This was a bug found by existing INSIST_🐔 tests! (The one I stupidly deleted because I didn't want to start fixing INSIST code 🤦.) If two indices share the same mapping, we still need to update fieldToMappedIndices. I've transferred the tests to use unmapped_fields=load instead.

@mouhc1ne

This PR adds support for resolving type conflicts with unmapped fields. Similar to how [union types](https://www.elastic.co/docs/reference/query-languages/esql/esql-multi-index#esql-multi-index-union-types) work, this is resolved using explicit casts, except we have an additional constraint that potentially unmapped fields are treated as `KEYWORD`. In particular, if a field is Partially Unmapped and when it is mapped it's a Non-Keyword (or PUNK, as per @mouhc1ne), we have a type conflict, which may itself be conflicted if the field is mapped to different fields (in addition to the unmapped `KEYWORD`). Resolves elastic#141912. Resolves elastic#142004.

GalLalouche and others added 30 commits January 21, 2026 23:04

Late materialization golden tests

5036b99

Remove old test file

84976f5

Fix failing test

3410841

Small changes

4066eb4

Merge branch 'main' into tests/late_golden

54390d1

CR: Add minimum random transport version

5635f49

CR: Add ESQL query to the output

84a15de

Remove local node reduction tests in some cases

06d4661

Merge branch 'main' into tests/late_golden

2d87993

More CR Fixes

f82a95a

Merge branch 'main' into tests/late_golden

c4cec77

Fix bug with synthetic names

a0eace3

Add name IDs

5fec441

Synthetic names

a976a88

Refactor

6a98ff7

Golden test start

dd3ef6b

Fix set issue

e0b20db

TEMP

6aeccac

Fix query.esql

d47e18b

TEMP

465826c

Added more tests

4c8ae63

Add multi index tests

eefa9cf

TEMP

7f69234

Push after potentially unmapped field hack

ac35237

Add field aliases test

66cb1ed

Stuffs

221c4fa

Added some nullify tests, fix CsvTests bug

9aaca40

[CI] Auto commit changes from spotless

7a3b810

alex-spies mentioned this pull request Mar 24, 2026

ESQL: add tests and identify not-yet working cases for unmapped_fields="load" #141911

Closed

25 tasks

alex-spies reviewed Mar 24, 2026

View reviewed changes

x-pack/plugin/esql/qa/testFixtures/src/main/resources/unmapped-load.csv-spec Show resolved Hide resolved

Skip load tests in CsvTests

f66e36e

alex-spies mentioned this pull request Mar 25, 2026

Disallow unmapped_fields=load with partial non-KEYWORD #144109

Open

GalLalouche added 8 commits March 26, 2026 12:57

Add null cases

c39104b

More CR reviews

9e3bbcb

Remove FIXME

452338a

Add tests

53f8e43

Add golden test coverage

f789d7a

More tests

6a45664

Add assertion

fae0da2

Fix nanos bug

cfee60d

GalLalouche commented Mar 26, 2026

View reviewed changes

[CI] Auto commit changes from spotless

18bf717

alex-spies mentioned this pull request Mar 26, 2026

ESQL: make unmapped_fields="load" automatically trigger for partially mapped KEYWORD fields, and allow projecting partially mapped non-KEYWORD fields #144228

Open

GalLalouche and others added 7 commits March 26, 2026 14:26

Merge branch 'main' into type_conflicts

7622b4c

Bump optional_fields to v3

dc33f91

Fix wrong counting when two indices have the same mapping

b6c2956

[CI] Auto commit changes from spotless

5a9b11e

Add more test coverage to double count bug

3166bc3

Rename test to avoid star expansions in existing tests

9f17be0

Fix yamlRestTest

9e3e7aa

GalLalouche commented Mar 26, 2026

View reviewed changes

GalLalouche added 3 commits March 26, 2026 20:20

Merge branch 'main' into type_conflicts

e995f13

Fix BWC

54acdb4

Merge branch 'main' into type_conflicts

ecb3e2e

GalLalouche merged commit c5b01cd into elastic:main Mar 27, 2026
37 checks passed

This was referenced Mar 27, 2026

ESQL: Fix mismatched error message on yaml test #145102

Closed

ESQL: Add timeseries X type conflict tests #145187

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ESQL: Type conflict resolution in unmapped-fields load#143693

ESQL: Type conflict resolution in unmapped-fields load#143693
GalLalouche merged 165 commits intoelastic:mainfrom
GalLalouche:type_conflicts

GalLalouche commented Mar 5, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

GalLalouche Mar 23, 2026

Uh oh!

GalLalouche Mar 23, 2026

Uh oh!

GalLalouche Mar 23, 2026

Uh oh!

GalLalouche Mar 23, 2026

Uh oh!

GalLalouche Mar 24, 2026

Uh oh!

GalLalouche Mar 25, 2026

Uh oh!

GalLalouche Mar 25, 2026

Uh oh!

Uh oh!

GalLalouche Mar 26, 2026

Uh oh!

GalLalouche Mar 26, 2026

Uh oh!

GalLalouche Mar 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

GalLalouche commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

GalLalouche commented Mar 5, 2026 •

edited

Loading